Information presentation apparatus, information presentation method, and non-transitory computer-readable storage medium

ABSTRACT

An information presentation apparatus includes: a signal input unit configured to input an image signal from a camera and a signal from a sensor; a signal-source-position estimation unit configured to estimate a position of a signal source, based on the signal from the sensor; a situation-expression extraction unit configured to extract situation expression indicating information expressing a situation outside a capture range of the camera, based on the signal from the sensor; a display-method determination unit configured to determine a display method of the situation expression on a captured image displayed by the image signal, based on a positional relation between the position of the signal source and the capture range of the camera; and a presentation unit configured to superimpose the situation expression onto the captured image, in accordance with the display method, and outputting the captured image superimposed with the situation expression.

This application is a National Stage Entry of PCT/JP2015/002939 filed onJun. 11, 2015, which claims priority from Japanese Patent Application2014-125866 filed on Jun. 19, 2014, the contents of all of which areincorporated herein by reference, in their entirety.

TECHNICAL FIELD

The present invention relates to a technology of presenting relatedinformation along with a captured image.

BACKGROUND ART

One would occasionally like to convey an atmosphere of a location wherehe or she is, to a person at a remote location such as a family livingapart and a colleague at a different branch. In such a case, part of anatmosphere of a space where he or she is can be conveyed by capturing animage of a surrounding environment and showing the image. However, acaptured image only is not able to completely convey the atmosphere ofthe space.

For example, a captured image is not able to convey an ambient sound. Anexample of a technology solving the problem includes a text insertionapparatus described in PTL 1. The text insertion apparatus generates animage with text by converting voice data obtained from an image withvoice into text information and inserting the converted text informationinto a background region or the like in a captured image. Thus, the textinsertion apparatus is able to convey an utterance content, an ambientsound, and the like accompanying the captured image, along with thecaptured image.

Further, there may be a case that a state and a temporal change of asubject cannot be grasped at a glance solely by a captured image only.For example, one feels “jumble up” when seeing a video image of a crowd,and feels “dead quiet” when seeing a video image of an office in whichno one is moving even though there are many people. However, it isdifficult to grasp such a state and a temporal change solely by onecaptured image only.

An example of a technology solving the problem includes asensibility-expression-word processing apparatus described in PTL 2. Thesensibility-expression-word processing apparatus stores in advance, acorrespondence relation between sensibility information and asensibility expression word, in a sensibility expression word database.The sensibility information is information indicating a situation of aplace, a state of an object, a temporal change of the place, and amovement of the object, being displayed in a captured image. Thesensibility expression word is a word expressing an impression of anatmosphere of the place and the object, imaged by a person seeing thecaptured image. The sensibility-expression-word processing apparatusanalyzes the captured image, calculates the sensibility information,selects a sensibility expression word associated with the calculatedsensibility information, and superimposes the selected sensibilityexpression word onto the captured image. Thus, thesensibility-expression-word processing apparatus is able to convey theatmosphere and the impression of the place and the object imaged in thecaptured image, along with the captured image.

CITATION LIST Patent Literature

PTL 1: Japanese Unexamined Patent Application Publication No. 2003-18462

PTL 2: International Application Publication No. WO 2012/070430

SUMMARY OF INVENTION Technical Problem

However, the aforementioned related arts have the following problems.

The text insertion apparatus described in PTL 1 converts auditoryinformation included in a captured video image signal (image with voice)into text information and inserts the information, and therefore is notable to sufficiently convey a situation outside a capture range at aplace including the capture range.

Further, the sensibility-expression-word processing apparatus describedin PTL 2 superimposes a sensibility expression word based on visualinformation included in a captured video image signal onto an image inthe video image signal, and therefore is not able to sufficiently conveya situation outside a capture range at a place including the capturerange.

The present invention is made in order to solve the aforementionedproblems, and an object thereof is to provide a technology capable of,when presenting a captured image, sufficiently conveying a situationoutside a capture range.

Solution to Problem

To achieve the purpose mentioned above, an information presentationapparatus of the present invention includes: signal input means forinputting an image signal from a camera and a signal from a sensor;signal-source-position estimation means for estimating a position of asignal source, based on the signal from the sensor, situation-expressionextraction means for extracting information expressing a situation(situation expression) outside a capture range of the camera, based onthe signal from the sensor; display-method determination means fordetermining a display method of the situation expression on a capturedimage displayed by the image signal, based on a positional relationbetween the position of the signal source and a capture range of thecamera, and presentation means for superimposing the situationexpression onto the captured image, in accordance with the displaymethod; and outputting the captured image superimposed with thesituation expression.

An information presentation system of the present invention includes acamera, a sensor, and the information presentation apparatus.

An information presentation method of the present invention includes:inputting an image signal of a camera and a signal from a sensor;estimating a position of a signal source, based on the signal from thesensor; extracting information expressing a situation (situationexpression) outside a capture range of the camera, based on the signalfrom the sensor;

determining a display method of the situation expression on a capturedimage displayed by the image signal, based on a positional relationbetween the position of the signal source and the capture range of thecamera; and superimposing the situation expression onto the capturedimage, in accordance with the display method, and outputting thecaptured image superimposed with the situation expression.

A storage medium of the present invention stores an informationpresentation program causing a computer apparatus to perform: a signalinput step of inputting an image signal of a camera and a signal from asensor; a signal-source-position estimation step of estimating aposition of a signal source, based on the signal from the sensor; asituation-expression extraction step of extracting informationexpressing a situation (situation expression) outside a capture range ofthe camera, based on the signal from the sensor; a display-methoddetermination step of determining a display method of the situationexpression on a captured image displayed by the image signal, based on apositional relation between the position of the signal source and thecapture range of the camera; and a presentation step of superimposingthe situation expression onto the captured image, in accordance with thedisplay method, and outputting the captured image superimposed with thesituation expression.

Advantageous Effects of Invention

The present invention is able to provide a technology capable of, whenpresenting a captured image, sufficiently conveying a situation outsidea capture range.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram illustrating a configuration of an informationpresentation system according to a first example embodiment of thepresent invention.

FIG. 2 is a hardware configuration of the information presentationsystem according to the first example embodiment of the presentinvention.

FIG. 3 is a flowchart illustrating an operation of the informationpresentation system according to the first example embodiment of thepresent invention.

FIG. 4 is a block diagram illustrating a configuration of an informationpresentation system according to a second example embodiment of thepresent invention.

FIG. 5 is a diagram illustrating an example of arrangement of a cameraand a sensor, according to the second example embodiment of the presentinvention.

FIG. 6 is a diagram illustrating an example of information stored in anoutside-capture-range sensor DB, according to the second exampleembodiment of the present invention.

FIG. 7 is a schematic diagram illustrating a positional relation betweena camera and a sensor, according to the second example embodiment of thepresent invention.

FIG. 8 is a diagram illustrating an example of information stored in asensor positional relation DB, according to the second exampleembodiment of the present invention.

FIG. 9 is a schematic diagram illustrating an example of a determinationtechnique of a display method, according to the second exampleembodiment of the present invention.

FIG. 10 is a schematic diagram illustrating another example of adetermination technique of a display method, according to the secondexample embodiment of the present invention.

FIG. 11 is a schematic diagram illustrating another example of adetermination technique of a display method, according to the secondexample embodiment of the present invention.

FIG. 12 is a schematic diagram illustrating another example of adetermination technique of a display method, according to the secondexample embodiment of the present invention.

FIG. 13 is a schematic diagram illustrating another example of adetermination technique of a display method, according to the secondexample embodiment of the present invention.

FIG. 14 is a schematic diagram illustrating another example of adetermination technique of a display method, according to the secondexample embodiment of the present invention.

FIG. 15 is a schematic diagram illustrating an example of asuperimposition technique of a situation expression, according to thesecond example embodiment of the present invention.

FIG. 16 is a schematic diagram illustrating another example of asuperimposition technique of a situation expression, according to thesecond example embodiment of the present invention.

FIG. 17 is a flowchart illustrating an operation of the informationpresentation system according to the second example embodiment of thepresent invention.

FIG. 18 is a diagram illustrating an example of information stored in asituation expression DB, according to the second example embodiment ofthe present invention.

FIG. 19 is a schematic diagram illustrating an example of a human regiondetected by a sensor, according to the second example embodiment of thepresent invention.

FIG. 20 is a schematic diagram illustrating an example of a displaymethod determined according to the second example embodiment of thepresent invention.

FIG. 21 is a diagram schematically illustrating an example of a capturedimage on which a situation expression is superimposed and presented,according to the second example embodiment of the present invention.

DESCRIPTION OF EMBODIMENTS

Example embodiments of the present invention will be described in detailbelow with reference to the drawings.

First Example Embodiment

FIG. 1 illustrates a configuration of an information presentation system1 according to a first example embodiment of the present invention.

In FIG. 1, the information presentation system 1 includes an informationpresentation apparatus 10, a camera 80, and a sensor 90. The informationpresentation apparatus 10 is communicably connected to the camera 80 andthe sensor 90, respectively.

The camera 80 is an apparatus capturing an image of a space andgenerating an image signal such as a moving image or a still image. Thecamera 80 is located at a location where an image of at least part ofthe space being an observation target of the information presentationsystem 1 can be captured.

While FIG. 1 illustrates one camera 80, a quantity of a camera includedin the information presentation system according to the presentinvention is not limited thereto.

The sensor 90 is an apparatus observing a situation of the space.Specifically, the situation of the space may include a situation of thespace itself (e.g. brightness, temperature, and humidity). Further, thesituation of the space may include a state of a target object (e.g. ahuman, an animal, and a thing) existing in the space. A specific exampleof the sensor 90 observing such the situation of the space includes, forexample, a camera (a camera different from the camera 80), a microphone,a depth sensor, a vibration sensor, an illuminometer, a hygrometer, anda thermometer. Further, the sensor 90 may be so configured that anobservation direction thereof changes like a pan-tilt camera. Further,the sensor 90 may be so configured that a position thereof changes likea sensor equipped on an automobile, an aircraft, or a robot.

Further, the sensor 90 outputs an observed signal to the informationpresentation apparatus 10. For example, when the sensor 90 is a camera,the sensor 90 outputs an image signal. For example, when the sensor 90is the microphone, the depth sensor, the vibration sensor, theilluminometer, the hygrometer, or the thermometer, the sensor 90 outputsa voice signal, a depth signal, a vibration signal, an illuminancesignal, a humidity signal, or a temperature signal.

The sensor 90 is located at a location where a situation of the spacebeing the observation target of the information presentation system 1can be observed. It is desirable that at least one sensor 90 be solocated that a situation outside a capture range of the camera 80 in thespace being the observation target of the information presentationsystem 1 can be observed. A sensor 90 so located that the situationoutside the capture range of the camera 80 can be observed may behereinafter referred to as an outside-capture-range sensor 90.

While FIG. 1 illustrates one sensor 90, a quantity of a sensor and aquantity of a type of a sensor, being included in the informationpresentation system according to the present invention, is not limitedthereto.

The information presentation apparatus 10 includes a signal input unit11, a signal-source-position estimation unit 12, a situation-expressionextraction unit 13, a display-method determination unit 14, and apresentation unit 15. FIG. 2 illustrates an example of a hardwareconfiguration of the information presentation apparatus 10. Theinformation presentation apparatus 10 is composed of a centralprocessing unit (CPU) 1001, a random access memory (RAM) 1002, a readonly memory (ROM) 1003, a storage apparatus 1004, a peripheral-equipmentconnection interface 1005, and a display apparatus 1006.

In this case, the signal input unit 11 is configured by theperipheral-equipment connection interface 1005, and the CPU 1001 readinga computer program and various types of data stored in the ROM 1003 andthe storage apparatus 1004 into the RAM 1002, and executing the programand the data. Further, the signal-source-position estimation unit 12,the situation-expression extraction unit 13, and the display-methoddetermination unit 14 are configured by the CPU 1001 reading thecomputer program and various types of data stored in the ROM 1003 andthe storage apparatus 1004 into the RAM 1002, and executing the programand the data. Further, the presentation unit 15 is configured by thedisplay apparatus 1006, and the CPU 1001 reading a computer program andvarious types of data stored in the ROM 1003 and the storage apparatus1004 into the RAM 1002, and executing the program and the data. Thehardware configuration of the information presentation apparatus 10 andthe respective functional blocks thereof are not limited to theaforementioned configuration.

The signal input unit 11 acquires an image signal of the camera 80 and asignal from the sensor 90 as input information. For example, when everysensor 90 is a camera, the signal input unit 11 acquires an image signalfrom the camera 80 and an image signal from each sensor 90. Further, itis assumed that each of the sensor 90 is a combination of a camera, amicrophone, a depth sensor, a vibration sensor, an illuminometer, ahygrometer, and a thermometer. In this case, the signal input unit 11acquires an image signal from the camera 80, and an image signal, avoice signal, a depth signal, a vibration signal, an illuminance signal,a humidity signal, and a temperature signal from the respective sensors90. Then, the signal input unit 11 feeds the signals acquired from thesensors 90 to the signal-source-position estimation unit 12 and thesituation-expression extraction unit 13. Further, the signal input unit11 may feed the image signal acquired from the camera 80 to thedisplay-method determination unit 14 and the presentation unit 15.Alternatively, the signal input unit 11 may temporarily store the imagesignal acquired from the camera 80 into the storage apparatus 1004, andfeed the signal to the display-method determination unit 14 and thepresentation unit 15 via the storage apparatus 1004.

The signal-source-position estimation unit 12 estimates a position of asignal source, in accordance with a signal from the sensor 90. Thesignal source is an origin of a signal being a source (factor) of asituation and an atmosphere of a space, or an observation target of thesensor 90, and includes, for example, origins of a sound, vibration,light, heat, and humidity in the space, and a still object and a movingobject existing in the space. For example, the signal-source-positionestimation unit 12 may estimate a position of the signal source byanalyzing the signals from the sensor 90. Further, for example, thesignal-source-position estimation unit 12 may apply a position of thesensor 90 outputting a signal as the position of the signal source.

The situation-expression extraction unit 13 extracts informationexpressing the situation (situation expression) outside the capturerange of the camera 80, based on a signal from the sensor 90. Forexample, the situation-expression extraction unit 13 may extract thesituation expression as a situation outside the capture range, based ona signal obtained from the sensor 90 located outside of the capturerange of the camera 80. The situation expression extracted based on thesignal obtained from the sensor 90 may also be hereinafter referred toas a situation expression of the sensor 90, a situation expressionregarding the signal source (of the sensor 90), a situation expressionof the signal source, or the like.

“Extraction” refers to determining and outputting the situationexpression matching the signal obtained from the sensor 90. Further, thesituation expression is information visually representing an atmosphereand a situation of the space. It is desirable that the situationexpression be visual information by which an observer is able toinstantaneously grasp the atmosphere and the situation of the space. Forexample, a text image expressing a text, and an image expressing asymbol such as a symbol image, an icon image, and a stamp image may beapplied as such the situation expression.

For example, when the signal from the sensor 90 indicates activation oflighting, the situation-expression extraction unit 13 may extract a textimage of a sensibility word “Flash,” as the situation expression. Inaddition, the situation expression may include a text image of asensibility word such as “Warming up” indicating a temperature rise byair conditioning, “Splash” indicating a humidity rise when water isspilled, “Striding” indicating how a person walks, and “Chattering”indicating how a person speaks. Further, the situation expression mayalso include a typical utterance such as “Eek” indicating a scream and“Hello” indicating a greeting, and a text image indicating a designationregarding a situation and an atmosphere such as “Lights out,” “Fire,”“Scream,” and “Anomaly.” Furthermore, the situation expression may alsoinclude a symbol image such as “!” indicating occurrence of an anomaly,and an icon with high conspicuity such as a red lamp indicating ananomaly. In addition, any information visually representing theatmosphere and the situation of the space is applicable as anothersituation expression.

The display-method determination unit 14 determines a display method ofthe situation expression on the captured image of the camera 80, basedon a positional relation between the position of the signal source andthe capture range of the camera 80. The captured image of the camera 80refers to an image displayed by the image signal supplied from thecamera 80 to the signal input unit 11. For example, when the imagesignal supplied from the camera 80 is a still image, the input stillimage is applied as the captured image. Further, for example, when theimage signal input from the camera 80 is a moving image, one or moreframe images constituting the input moving image is applied as thecaptured image.

For example, the display-method determination unit 14 may determine adisplay position on the captured image as the display method of thesituation expression. Specifically, the display-method determinationunit 14 may determine the display position of the situation expressionregarding the signal source, in accordance with a relative direction ofthe position of the signal source with respect to the capture range ofthe camera 80. In addition, the display-method determination unit 14 maydetermine the display position of the situation expression regarding thesignal source, in accordance with a distance between the capture rangeand the position of the signal source.

The presentation unit 15 superimposes the situation expression onto thecaptured image, in accordance with a display method determined by thedisplay-method determination unit 14, and outputs the captured imagesuperimposed with the situation expression. For example, thepresentation unit 15 superimposes the situation expression onto thedisplay position determined by the display-method determination unit 14.Then, the presentation unit 15 may output the captured imagesuperimposed with the situation expression to the display apparatus1006. Further, for example, the presentation unit 15 may output thecaptured image superimposed with the situation expression to the storageapparatus 1004. Further, for example, the presentation unit 15 mayoutput the captured image superimposed with the situation expression toanother apparatus connected via a network interface (unillustrated) orthe like.

An operation of the information presentation system 1 configured asdescribed above will be described with reference to FIG. 3.

The signal input unit 11 acquires an image signal from the camera 80 anda signal from the sensor 90 as inputs (Step A1).

The signal-source-position estimation unit 12 estimates a position of asignal source, in accordance with the signal from the sensor 90 (StepA2). For example, as described above, the signal-source-positionestimation unit 12 may apply a position of the sensor 90 as the positionof the signal source, or may estimate the position of the signal sourceby analyzing the signal.

The situation-expression extraction unit 13 extracts a situationexpression outside the capture range of the camera 80, in accordancewith the signal from the sensor 90 (Step A3). For example, as describedabove, the situation-expression extraction unit 13 may extract thesituation expression outside the capture range, in accordance with thesignal obtained from the outside-capture-range sensor 90 of the camera80.

The display-method determination unit 14 determines the display methodon the captured image of the camera 80, with respect to the situationexpression extracted in Step A3 (Step A4). For example, as describedabove, the display-method determination unit 14 may determine thedisplay position of the situation expression on the captured image, inaccordance with the relative direction and the distance of the positionof the signal source with respect to the capture range of the camera 80.

The presentation unit 15 superimposes each situation expressionextracted in Step A3 onto the captured image displayed by the imagesignal of the camera 80 input in Step A1, in accordance with the displaymethod determined in Step A4, and outputs the captured imagesuperimposed with the situation expression (Step A5).

The above concludes the operation of the information presentation system1.

Next, an effect of the first example embodiment of the present inventionwill be described.

When presenting a captured image, the information presentation systemaccording to the first example embodiment of the present invention isable to more sufficiently convey a situation outside a capture range.

The reason is that the signal input unit acquires an image signal fromthe camera and a signal from a sensor as inputs, thesignal-source-position estimation unit estimates a position of a signalsource, in accordance with the signal from the sensor, and thesituation-expression extraction unit extracts a situation expressionoutside a capture range, based on the signal from the sensor. Further,an additional reason is that the display-method determination unitdetermines a display method on a captured image, based on a positionalrelation between the position of the signal source and the capturerange, and the presentation unit superimposes the situation expressiononto the captured image, in accordance with the determined displaymethod, and outputs the captured image superimposed with the situationexpression.

Thus, the present example embodiment superimposes the situationexpression outside the capture range onto the captured image by thedisplay method based on the positional relation between the capturerange and the position of the signal source, and therefore is able toconvey a situation occurring outside the capture range and anapproximate location of the occurrence, along with the captured image.

Second Example Embodiment

Next, a second example embodiment according to the present inventionwill be described with reference to drawings. In the respective drawingsreferred to in the description of the present example embodiment, a samecomponent as and a step operating similarly to the first exampleembodiment according to the present invention are respectively givensame reference signs as the first example embodiment, and detaileddescription thereof is omitted in the present example embodiment.

FIG. 4 illustrates a configuration of an information presentation system2 according to the second example embodiment of the present invention.

In FIG. 4, the information presentation system 2 differs from theinformation presentation system 1 according to the first exampleembodiment of the present invention in including an informationpresentation apparatus 20 instead of the information presentationapparatus 10. The information presentation apparatus 20 is communicablyconnected to the camera 80 and the sensor 90, respectively. While FIG. 4illustrates one each of the camera 80 and the sensor 90, a quantity ofthe camera, a quantity of the sensor, and a quantity of a type of thesensor, being included in the information presentation system accordingto the present invention is not limited thereto.

The information presentation apparatus 20 differs from the informationpresentation apparatus 10 according to the first example embodiment ofthe present invention in including a signal-source-position estimationunit 22 instead of the signal-source-position estimation unit 12, and asituation-expression extraction unit 23 instead of thesituation-expression extraction unit 13. The information presentationapparatus 20 further differs from the information presentation apparatus10 in including a display-method determination unit 24 instead of thedisplay-method determination unit 14, and a presentation unit 25 insteadof the presentation unit 15.

Nearly similarly to the signal-source-position estimation unit 12according to the first example embodiment of the present invention, thesignal-source-position estimation unit 22 estimates a position of asignal source, based on a signal from the sensor 90. Thesignal-source-position estimation unit 22 according to the presentexample embodiment in particular estimates the position of the signalsource as follows, in respective cases of the signal source being asound, a moving object, and a human.

When the signal source is a sound, the signal-source-position estimationunit 22 is able to estimate the position of the signal source asfollows. It is assumed that a plurality of microphones are located assensors 90. In this case, the signal-source-position estimation unit 22compares frequency components and compares energy, of respective soundsignals acquired from the plurality of microphones. Then, thesignal-source-position estimation unit 22 may determine a position of amicrophone (sensor 90) capturing a sound signal having the highestenergy out of sounds composed of a same type of a frequency component,to be a position of the source of the sound signal.

Further, when the signal source is a moving object, thesignal-source-position estimation unit 22 is able to estimate a positionof the signal source as follows. It is assumed that a camera and a depthsensor are located as sensors 90 at a nearly same position in a nearlysame observation direction. In this case, the signal-source-positionestimation unit 22 observes point group data in a three-dimensionalspace for a certain period of time, in accordance with an image signaland a depth signal obtained from the camera and the depth sensor. Then,the signal-source-position estimation unit 22 detects a positional shiftwith respect to a point group on an image region demarcated inaccordance with an image feature such as a color and an edge. Then, thesignal-source-position estimation unit 22 may estimate a centercoordinate out of the point group the positional shift of which isdetected, in a coordinate system based on the camera and the depthsensor, as a position of the moving object.

Furthermore, when the signal source is a human, thesignal-source-position estimation unit 22 is able to estimate a positionof the signal source as follows. It is assumed that a camera is locatedas the sensor 90. In this case, the signal-source-position estimationunit 22 detects a human region, in accordance with an image signal fromthe camera as the sensor 90. Then, the signal-source-position estimationunit 22 may estimate a position of the human being the signal source, byuse of a table or a conversion function associating in advance aposition on an image obtained from the camera as the sensor 90 with aposition in the space. Further, in this case, as a detection techniqueof the human region, the signal-source-position estimation unit 22 mayuse a background subtraction method detecting a region different from aspace image created in advance as the human region. Alternatively, as adetection technique of the human region, the signal-source-positionestimation unit 22 may use template matching scanning an input imagewith a template image of a human region created in advance, anddetecting a region having a similar image characteristic to the humanregion. Another known technology may be adopted in detection processingof a human region, without being limited to the above.

While an example of an estimation technique of a position of a signalsource has been cited for each case of the signal source being a sound,a moving object, and a human, as described above, thesignal-source-position estimation unit 22 may use another technology forestimating a position of a signal source with respect to the varioustypes of signal sources. Further, the signal-source-position estimationunit 22 may use various known technologies for estimating a position ofa signal source with respect to signal sources other than theaforementioned types.

Further, when an observation direction or a position of the sensor 90changes, the signal-source-position estimation unit 22 estimates aposition of a signal source in view of the change in observationdirection and position. In this case, the sensor 90 may be equipped witha position detection sensor or a direction detection sensor in advance.The position detection sensor may be a global positioning system (GPS)or another sensor capable of detecting the position. Further, thedirection detection sensor may be a gyro-sensor or another sensorcapable of detecting the direction. Then, in accordance with informationobtained from the position detection sensor or the direction detectionsensor, the signal-source-position estimation unit 22 may specify aposition and an observation direction of the sensor in a worldcoordinate system, and estimate a position of the signal source by useof the specified information about the position and the observationdirection.

The situation-expression extraction unit 23 extracts the situationexpression outside the capture range of the camera 80 with respect to asignal from the sensor 90, by use of a situation expression database(DB) and an outside-capture-range sensor DB. Additionally, thesituation-expression extraction unit 23 may extract the situationexpression within the capture range of the camera 80.

The situation expression DB stores information associating informationabout a signal obtainable from the sensor 90 with the situationexpression. The situation expression DB is prepared in advance andstored in the storage apparatus 1004 or the like. The information abouta signal obtainable from the sensor 90 may be a condition related to afeature amount of a signal obtainable from the sensor 90. Further, theinformation about a signal obtainable from the sensor 90 may be acondition related to a temporal change of a signal obtainable from thesensor 90 or a feature amount of the signal.

Further, the situation expression DB may associate the situationexpression with a condition combining signals from a plurality ofsensors 90, and store the situation expression. In this case, thesituation-expression extraction unit 23 may extract the situationexpression associated with a condition met by a combination of signalsrespectively input from the plurality of sensors 90. For example, thesituation-expression extraction unit 23 is able to extract the situationexpression indicating a fire when a combination of a temperature signalfrom a thermometer and an illuminance signal from an illuminometer meetsa condition indicating rise in both temperature and illuminance.Further, the situation-expression extraction unit 23 is able to extractthe situation expression indicating occurrence of an incident when acombination of a voice signal from a microphone and an image signal froma camera as the sensor 90 meets a condition indicating a voice with acertain sound level or higher, and movement of a certain number ofpeople or more.

The outside-capture-range sensor DB stores information indicatingwhether or not the sensor 90 is the outside-capture-range sensor 90 ofthe camera 80. The outside-capture-range sensor 90 may be defined as thesensor 90 having an observation region in which a region not included inthe capture range of the camera 80 exists. In this case, theoutside-capture-range sensor 90 is defined regardless of whether anlocated position of the sensor 90 is within the capture range of thecamera 80 or outside the capture range.

For example, when the information presentation system 2 includes theplurality of cameras 80 and the plurality of sensors 90, theoutside-capture-range sensor DB may store information indicating whetheror not each sensor 90 is the outside-capture-range sensor 90 withrespect to each camera 80. Further, when the information presentationsystem 2 includes the plurality of cameras 80, each camera 80 may beapplied as the camera 80 according to the present example embodiment,and, for each camera 80, another camera 80 may be applied as the sensor90 according to the present example embodiment. In this case, theoutside-capture-range sensor DB may store information indicating, foreach camera 80, whether or not another camera 80 is theoutside-capture-range sensor 90.

For example, it is assumed that the information presentation system 2includes, in an observation space thereof, three cameras 1 to 3, threemicrophones 1 to 3, and two thermometers 1 and 2, as illustrated in FIG.5. FIG. 5 illustrates a desk layout in an office. In FIG. 5, eachrectangle represents a desk, and a space between rectangles representsan aisle through which a person is able to pass.

In this case, the outside-capture-range sensor DB may store informationas listed in FIG. 6. For example, in FIG. 6, the first row indicatesthat the cameras 2 and 3, the microphones 1 to 3, and the thermometer 2are outside-capture-range sensors 90 of the camera 1, and thethermometer 1 is not the outside-capture-range sensor 90 of the camera 1(a sensor 90 within the capture range). Thus, the outside-capture-rangesensor DB may have a format storing information representing theoutside-capture-range sensor 90 by “∘,” and storing informationrepresenting the sensor 90 within the capture range by “x,” for eachcamera 80. By referring to such the outside-capture-range sensor DB, thesituation-expression extraction unit 23 may specify theoutside-capture-range sensor 90 of a given camera 80, and extract asituation expression outside the capture range of the camera 80 by useof a signal from the specified sensor 90. The outside-capture-rangesensor DB may store information in another format, without being limitedto the format in FIG. 6.

Further, when a position or an observation direction of the sensor 90changes, the situation-expression extraction unit 23 may appropriatelyupdate the outside-capture-range sensor DB. For example, when the sensor90 is equipped with a position detection sensor and a directiondetection sensor, as described above, the situation-expressionextraction unit 23 acquires information held by the position detectionsensor and the direction detection sensor, along with a signal from thesensor 90. Then, the situation-expression extraction unit 23 may updatethe outside-capture-range sensor DB, in accordance with whether or not aregion not included in the capture range of the camera 80 exists in anobservation region of the sensor 90, being calculated in accordance withinformation held by the position detection sensor and the directiondetection sensor.

Further, as described above, the situation-expression extraction unit 23may extract a situation expression within the capture range of thecamera 80, in addition to extracting a situation expression outside thecapture range of the camera 80. In this case, the situation-expressionextraction unit 23 may extract a situation expression from the situationexpression DB, corresponding to a signal input to the signal input unit11 from a sensor 90 predetermined to be within the capture range of thecamera 80, and take the situation expression as a situation expressionwithin the capture range of the camera 80. The situation-expressionextraction unit 23 is able to specify the sensor 90 within the capturerange of the camera 80 by referring to the outside-capture-range sensorDB.

The display-method determination unit 24 determines a display method ofa situation expression on a captured image of the camera 80, based on apositional relation between a position of the signal source and thecapture range of the camera 80. At this time, the display-methoddetermination unit 24 determines a detailed display method as follows,in addition to determining the display position of the situationexpression on the captured image, similarly to the display-methoddetermination unit 14 according to the first example embodiment of thepresent invention.

For example, the display-method determination unit 24 may determine ashape or a size of the situation expression on the captured image. Theshape may include, for example, a shape of a surrounding frame of a textimage as the situation expression, and a shape transforming the textimage itself. Further, as the display method, the display-methoddetermination unit 24 may determine a transmissivity when superimposingthe situation expression onto the captured image. As the display method,the display-method determination unit 24 may determine a movement of thesituation expression on the captured image. It is assumed that themovement of the situation expression refers to a temporal change (e.g.blinking, moving, and a change rate thereof) of the display method ofthe situation expression. The movement of the situation expression isnot limited to blinking, moving, and a change rate thereof, and may beanother movement expressible on the captured image. Further, as thedisplay method, the display-method determination unit 24 may determinewhether or not to superimpose a balloon and an effect line accompanyinga situation expression, for indicating a direction of a signal sourcestarting from a position of the camera 80.

Further, the display-method determination unit 24 may determine adisplay method of a situation expression, in view of a moving speed anda moving direction of a signal source, or importance (a degree ofinterest and urgency for an observer) of the signal source. When atleast part of a display region of a situation expression overlaps with adisplay region of another situation expression, the display-methoddetermination unit 24 may determine an overlapping order (an order offoreground/background) of the situation expressions on a captured image.For example, the display-method determination unit 24 may determine theoverlapping order in accordance with distances between the camera 80 andthe sensors 90 being origins of the respective extracted situationexpressions. Alternatively, the display-method determination unit 24 maydetermine the overlapping order of the situation expressions, based onimportance preset to the respective situation expressions, or importancepreset to the sensors 90 being origins of the respective extractedsituation expressions.

As described above, the display-method determination unit 24 maydetermine a display method capable of conveying, to an observer, aposition of a signal source outside a capture range, and a situation andan atmosphere of a space caused by the signal source.

Further, the display-method determination unit 24 may determine displaymethods as described above, by use of the sensor positional relation DB.The sensor positional relation DB stores information indicating thepositional relation between the capture range of the camera 80 and theposition of the sensor 90. Such the sensor positional relation DB isprepared in advance, and is stored in the storage apparatus 1004 or thelike. It is assumed that the positional relation stored in the sensorpositional relation DB is pre-calculated or pre-measured.

For example, the sensor positional relation DB may store a positioncoordinate of the sensor 90 in a coordinate system based on the camera80.

Alternatively, the sensor positional relation DB may store a positioncoordinate of any point included in an observation region of the sensor90 in a coordinate system based on the camera 80. For example, when thesensor 90 is a camera, the sensor positional relation DB may store aposition coordinate of any point on an optical axis of the camera beingthe sensor 90 in the coordinate system based on the camera 80. Thecoordinate system based on the camera 80 may be a two-dimensionalcoordinate indicating a position on the captured image of the camera 80.The two-dimensional coordinate indicating the position on the capturedimage of the camera may also be hereinafter referred to as a cameraimage coordinate.

For example, it is assumed that, with respect to the camera 2illustrated in FIG. 5, a position coordinate of another sensor 90 in acamera image coordinate system of the camera 2 is indicated as FIG. 7.In this case, the sensor positional relation DB stores information aslisted in FIG. 8.

When a position of the sensor 90 is applied as a position of a signalsource, the display-method determination unit 24 may refer to the sensorpositional relation DB and determine the display method, by use of aposition coordinate of the sensor 90 in a coordinate system based on thecamera 80. When the position of the signal source based on the positionof the sensor 90 is estimated as a position of a signal source, thedisplay-method determination unit 24 calculates a positional relationbetween the position of the signal source and the capture range asfollows. Specifically, the display-method determination unit 24 refersto the sensor positional relation DB, and obtains a position coordinateof the sensor 90 in the coordinate system based on the camera 80. Then,the display-method determination unit 24 may calculate a positioncoordinate of the signal source in the coordinate system based on thecamera 80, in accordance with the position coordinate of the sensor 90in the coordinate system based on the camera 80, and the position of thesignal source based on the position of the sensor 90.

As described above, the display-method determination unit 24 maycalculate the position coordinate of the signal source in the cameraimage coordinate system of the camera 80, in accordance with a positionof the signal source estimated by the signal-source-position estimationunit 22, and determine a display method of a situation expression on acaptured image, based on the calculated position coordinate.

In addition, the sensor positional relation DB may store a relativethree-dimensional position coordinate of the sensor 90 with respect tothe camera 80 in a three-dimensional space (e.g. a world coordinatesystem). Further, the sensor positional relation DB may storeinformation indicating a relative direction of the sensor 90 withrespect to the camera 80. The information indicating the relativedirection may include, for example, leftward, rightward, above, andbelow.

Further, when a position and an observation direction of the camera 80or the sensor 90 change, the display-method determination unit 24 maycalculate a positional relation between the camera 80 and the sensor 90,in accordance with the change in position and observation direction, andupdate the sensor positional relation DB. For example, when the camera80 and the sensor 90 are respectively equipped with a position detectionsensor and a direction detection sensor, the display-methoddetermination unit 24 is able to update the sensor positional relationDB, in accordance with signals from the position detection sensor andthe direction detection sensor.

The display-method determination unit 24 may adopt, for example, thetechniques cited below as specific examples of determination processingof a display method.

(103-A) Display-method determination Technique Based on PositionCoordinate of Sensor 90

It is assumed that a position of the sensor 90 itself is applied by thesignal-source-position estimation unit 22 as a position of a signalsource. It is further assumed that the sensor positional relation DBstores a position coordinate of the sensor 90 in a camera imagecoordinate system of the camera 80. In this case, the display-methoddetermination unit 24 may determine a display method as follows.

First, as illustrated in FIG. 9, the display-method determination unit24 determines an intersection of a line connecting a center (Cx,Cy) of acaptured image of the camera 80 and a coordinate (−Sx,Sy) of the sensor90, and a boundary of the captured image of the camera 80, in a cameraimage coordinate system of the camera 80, as a display reference pointof a situation expression. Then, the display-method determination unit24 determines a region with a predetermined range based on the displayreference point in a preset displayable region in the captured image, asa tentative display position. The displayable region is predetermined asa region onto which the situation expression can be superimposed. Forexample, the displayable region may be determined as a region on thecaptured image corresponding to an angle range predetermined within anangle of view of the camera 80. As illustrated in FIG. 9, such adisplayable region may be determined as, for example, a region definedby a length Ax in an x-direction and a length Ay in a y-direction,inward from the boundary of the captured image (a shaded region in FIG.9). Then, when there is no overlap with the display position determinedwith respect to another signal source, the display-method determinationunit 24 employs and determines the tentative display position as thedisplay position for the situation expression with respect to the signalsource. Further, with respect to the situation expression of each signalsource, the display-method determination unit 24 may adjust a size, atransmissivity, a distance from the display reference point, and thelike, in accordance with a distance between the image center of thecamera 80 and a coordinate of the signal source. For example, it isassumed, as illustrated in FIG. 10, that a distance between the imagecenter (Cx,Cy) of the camera 80 and a coordinate (−Sx,Sy) of a sensor 1is shorter than a distance between the image center and a coordinate(−S′x,S′y) of a sensor 2. In this case, the display-method determinationunit 24 determines respective display positions of situation expressionsso that a distance between a situation expression of the sensor 1 and adisplay reference point thereof is shorter than a distance between asituation expression of the sensor 2 and a display reference pointthereof. Further, in this case, the display-method determination unit 24determines respective sizes of the situation expressions so that thesituation expression of the sensor 1 is larger than the situationexpression of the sensor 2. When the situation expression displays text,the display-method determination unit 24 may preset, as a situationexpression size, a minimum size for an observer to be able to read thetext, and determine a size of the situation expression to be larger thanor equal to the preset size.

(103-B) Display-method determination Technique Based on PositionCoordinate of Signal Source

It is assumed that a position of a signal source based on a position ofthe sensor 90 is estimated by the signal-source-position estimation unit22 as the position of the signal source. It is further assumed that thesensor positional relation DB stores a position coordinate of the sensor90 in a camera image coordinate system of the camera 80. Further, as anexample, a case that the camera 2 in FIG. 5 is designated as the camera80, and the camera 1 as the sensor 90, is considered. In this case, thedisplay-method determination unit 24 may determine a display method asfollows.

As illustrated in FIG. 11, in a captured image of the camera 1 as thesensor 90, a human region is detected, and a center coordinate (P1x,P1y)of the human region is estimated with respect to a center (C1x,C1y) ofthe captured image of the camera 1. Then, the display-methoddetermination unit 24 obtains a position coordinate (−c21x,c21y) of thecamera 1 as the sensor 90 in a camera image coordinate system of thecamera 2 as the camera 80, from the sensor positional relation DB inFIG. 8. Then, as illustrated in FIG. 12, the display-methoddetermination unit 24 calculates a center coordinate (−p21x,p21y) of thehuman region in the camera image coordinate system of the camera 2. Thecenter coordinate (−p21x,p21y) of the human region in the camera imagecoordinate system of the camera 2 can be calculated from the positioncoordinate (−c21x,c21y) of the camera 1 in the camera image coordinatesystem of the camera 2 and the center coordinate (P1x,P1y) of the humanregion with respect to the image center of the camera 1. Then, thedisplay-method determination unit 24 determines an intersection of aline connecting a captured image center (C2x,C2y) of the camera 2 andthe center coordinate (−p21x,p21y) of the human region, and a frame of acaptured image of the camera 2, as a display reference point. Then, thedisplay-method determination unit 24 determines a region within apredetermined range from the display reference point as the displayposition of the situation expression regarding the human.

When a movement of the human region is detected in an image signal fromthe camera 1 being the sensor 90, the display-method determination unit24 may determine a movement of a situation expression, in accordancewith the movement of the human region. For example, when a humandisplayed in the human region moving in a direction toward the camera 2being the camera 80 is detected, the display-method determination unit24 may determine a temporal change of a transmissivity of the situationexpression so that the transmissivity is reduced as the human approachesthe camera 2. Further, in this case, the display-method determinationunit 24 may set a flag indicating that the transmissivity of thesituation expression is to be reduced from the transmissivity thereofwhen superimposed onto the captured image in an immediately precedingframe.

(103-C) Display-Method Determination Technique Based on RelativeDirection (e.g. “Above,” “Below,” “Leftward,” and “Rightward”)

It is assumed that the sensor positional relation DB stores “above,”“below,” “rightward,” and “leftward” as information indicating arelative direction of the sensor 90 with respect to the camera 80. As aposition of a signal source, either one of a position of the sensor 90itself, and a case of using a relative position of the signal sourcewith respect to the position of the sensor 90, may be applied. In thiscase, the display-method determination unit 24 may determine a displaymethod as follows.

As illustrated in FIG. 13, the display-method determination unit 24predetermines a point a as a display reference point with respect to asituation expression of the sensor 90, a relative position of which withrespect to the camera 80 is stored as “leftward.” Then, thedisplay-method determination unit 24 determines a region within apredetermined range from the display reference point (e.g. a rectangularregion A) as the display position of the situation expression.Similarly, the display-method determination unit 24 predetermines pointsb, c, and d as display reference points with respect to situationexpressions of sensors 90, relative positions of which with respect tothe camera 80 are stored as “above,” “rightward,” and “below,”respectively. Then, the display-method determination unit 24 determinesregions within predetermined ranges from the respective displayreference points (e.g. rectangular regions B, C, and D) as displaypositions of the situation expressions.

(103-D) Display-Method Determination Technique Based onThree-Dimensional Position Coordinate

It is assumed that the sensor positional relation DB stores a relativeposition coordinate of the sensor 90 with respect to a position of thecamera 80 in a three-dimensional coordinate system. As a position of asignal source, either one of a position of the sensor 90 itself, and acase of using a relative position of the signal source with respect tothe position of the sensor 90, may be applied. It is assumed, however,that the position of the signal source is estimated as athree-dimensional position coordinate. In this case, the display-methoddetermination unit 24 may determine a display method as follows.

In this case, as illustrated in FIG. 14, the display-methoddetermination unit 24 projects a line connecting an image center(Cx,Cy,Cz) of the camera 80 and a position coordinate (−Sx,−Sy,Sz) ofthe signal source, in a world coordinate system, onto a projection plane(captured image) of the camera 80. Then, the display-methoddetermination unit 24 sets an intersection of the projection line and aframe of the projection plane of the camera 80 as a display referencepoint. The display-method determination unit 24 determines a regionwithin a predetermined range from the display reference point as thedisplay position of the situation expression regarding the signalsource.

Further, when any one of the aforementioned display-method determinationtechniques (103-A) to (103-D) is adopted, the display-methoddetermination unit 24 may further determine a size of a situationexpression, in accordance with importance set to the sensor 90.Specifically, the display-method determination unit 24 may determine asize of a situation expression so that a situation expression of thesensor 90 of higher importance has a larger size. For example, it isassumed that sensors 90 such as a thermometer and a hygrometer are setwith higher importance than another sensor 90. In this case, thedisplay-method determination unit 24 is able to determine to superimposea situation expression in a larger size with respect to situationexpressions regarding the sensors 90 being a thermometer and ahygrometer (expressions indicating an abnormal situation of a space suchas occurrence of a fire and a water spill). Consequently, an observer isable to readily notice a situation of higher importance.

Alternatively, the display-method determination unit 24 may determine asize of an extracted situation expression, in accordance with importanceset to the situation expression itself, so that a situation expressionof higher importance has a larger display region. For example, it isassumed that a specific situation expression (e.g. a situationexpression indicating a state of emergency such as “Fire” and “Scream”)is set with higher importance than another situation expression. In thiscase, the display-method determination unit 24 is able to determine tosuperimpose a situation expression regarding such a state of emergencyin a larger size. Consequently, an observer is able to readily noticeoccurrence of a state of emergency.

Further, the display-method determination unit 24 may hold a flagindicating whether or not to display a balloon and an effect linespecifying a direction of a signal source as a display method.

Nearly similarly to the display-method determination unit 14 accordingto the first example embodiment of the present invention, thepresentation unit 25 superimposes a situation expression onto a capturedimage, in accordance to a display method determined by thedisplay-method determination unit 24, and outputs the captured imagesuperimposed with the situation expression. Further, the presentationunit 25 according to the present example embodiment is able to adopt,for example, the techniques cited below as presentation processing of asituation expression.

(104-A) Technique of Superimposing Situation Expression for CertainPeriod of Time

In this case, the presentation unit 25 superimposes a situationexpression onto a same position on a captured image of the camera 80 fora predetermined period of time. For example, it is assumed that an imagesignal of the camera 80 is input at 30 frames/sec, and a situationexpression is extracted once in every 10 seconds. In this case, thepresentation unit 25 presents, in a superimposed manner, a situationexpression extracted at a time t onto an image of 300 frames input for10 seconds from the time t, by a same display method.

(104-B) Technique of Superimposing with Balloon and Effect Line

It is assumed that, in a display method determined by the display-methoddetermination unit 24, a flag for displaying an effect line and aballoon is set. In this case, the presentation unit 25 superimposes asituation expression onto a captured image with a balloon and an effectline. For example, as illustrated in FIG. 15, the presentation unit 25may display a balloon and an effect line starting from a displayreference point. Consequently, an observer is able to intuitively see adisplay reference point indicating a position of a source of anatmosphere and a situation of an observation space.

When an overlapping order of a plurality of situation expressions isdetermined by the display-method determination unit 24, the presentationunit 25 may display the situation expressions, in accordance with theoverlapping order. For example, as illustrated in FIG. 16, thepresentation unit 25 may display situation expressions, in accordancewith an overlapping order determined by a distance between the sensor 90and the camera 80. Specifically, the presentation unit 25 maysuperimpose a situation expression of the sensor 90 closer to the camera80 in front of the situation expression of a farther sensor 90.Consequently, an observer is able to intuitively see a positionalrelation between a plurality of signal sources outside a capture range.

(104-C) Technique of Superimposing Situation Expression Extracted fromSensor 90 within Predetermined Range Only

In this case, the presentation unit 25 may superimpose a situationexpression extracted from the sensor 90 observing a predeterminedobservation range, out of one or more sensors 90 included in theinformation presentation system 2. For example, the sensor 90 observingthe predetermined observation range may be a sensor within apredetermined distance range from the camera 80. Consequently, anobserver is able to efficiently check an atmosphere and a situation of aspecific part particularly preferred to be checked, outside a capturerange.

(104-D) Technique of Superimposing Based on Change in Atmosphere andSituation

In this case, the presentation unit 25 may superimpose a situationexpression, while changing a size, a transmissivity, a position, and thelike thereof for a certain period of time. For example, when a movinghuman as a signal source is detected by analyzing a signal from thesensor 90, the presentation unit 25 may superimpose a situationexpression, while changing a size and a position thereof, for a certainperiod of time, within a range of the display position determined by thedisplay-method determination unit 24. When a human as a signal sourcemoving toward the camera 80 is detected by analyzing a signal from thesensor 90, the presentation unit 25 may superimpose a situationexpression, while reducing a transmissivity thereof, for a certainperiod of time. When a matter of urgency such as a sharp rise intemperature and humidity is detected by analyzing a signal from thesensor 90, the presentation unit 25 may switch between superimposing andnot superimposing a situation expression at certain intervals to blinkthe situation expression. Further, the presentation unit 25 may change(e.g. proportionate) a change rate of a size and a position, a changerate of a transmissivity, an on-off switching speed of superimposition,and the like, with respect to a situation expression, in accordance witha temporal change of a signal from the sensor 90 such as a moving speedof a human and a rising speed of temperature and humidity. Consequently,the presentation unit 25 is able to convey to an observer not only aposition of a source but also a degree of change thereof. Additionally,the presentation unit 25 is able to readily convey to an observer how anevent outside a capture range (or within the range) propagates to asituation within the capture range (or outside the range). Propagationof an event or a situation refers to, for example, a state of a humanentering a capture range of the camera 80, a state of a human exitingthe capture range, and a state of occurrence of buzzing within thecaptured image in response to buzzing outside the capture range.

(104-E) Technique of Superimposing Situation Expression within CaptureRange Along with Situation Expression Outside Capture Range

In this case, the presentation unit 25 may superimpose a situationexpression within a capture range of the camera 80 onto a capturedimage, in addition to a situation expression outside the capture range,by use of a display method determined by the display-methoddetermination unit 24. Consequently, the presentation unit 25 is able toconvey to an observer whether or not atmospheres and situations withinthe capture range and outside the capture range are similar.

(104-F) Technique of Superimposing Situation Outside Capture RangeDifferent from Situation within Capture Range

In this case, out of one or more situation expressions outside a capturerange of the camera 80, the presentation unit 25 may superimpose asituation expression different from a situation expression within thecapture range. Consequently, the presentation unit 25 is able to conveyto an observer an approximate location outside the capture range where asituation different from a situation that can be viewed and heard viathe camera 80 occurs.

(104-G) Technique of Superimposing Based on Change in Position andDirection of Camera 80

In this case, the presentation unit 25 may display, in a superimposedmanner, a situation expression, in accordance with a situationexpression successively extracted and a display method successivelydetermined, in response to a change in position and observationdirection of the camera 80. For example, it is assumed that an imagesignal of the camera 80 is input at 30 frames/sec, and a situationexpression and a display method thereof corresponding to a change inposition and observation direction of the camera 80 are extracted anddetermined once in every three seconds, respectively. In this case, thepresentation unit 25 presents, in a superimposed manner, a situationexpression extracted at a time t onto an image of 90 frames input forthree seconds from the time t, in accordance with a display methoddetermined at the time t. Consequently, even when a position and anobservation direction of the camera 80 change, the presentation unit 25is able to convey to an observer a situation outside a capture range, inaccordance with the position and the observation direction.

An operation of the information presentation system 2 configured asdescribed above will be described with reference to FIG. 17.

Similarly to the first example embodiment of the present invention, thesignal input unit 11 executes Step A1 and acquires an image signal fromthe camera 80 and a signal from the sensor 90 as inputs.

Based on the signal from the sensor 90, the signal-source-positionestimation unit 22 estimates a position of the sensor 90 itself or aposition obtained by analyzing the signal from the sensor 90 as aposition of the signal source (Step B2).

For example, with respect to a voice signal, an illuminance signal, anda temperature signal, the signal-source-position estimation unit 22estimates a position of the sensor 90 outputting the signal as aposition of the signal source. Further, with respect to an image signal,the signal-source-position estimation unit 22 analyzes and estimates aposition of a human region detected from the signal.

The situation-expression extraction unit 23 extracts a situationexpression outside a capture range of the camera 80 by referring to theoutside-capture-range sensor DB and the situation expression DB.Additionally, the situation-expression extraction unit 23 may extract asituation expression within the capture range of the camera 80 (StepB3).

Specifically, the situation-expression extraction unit 23 analyzes asignal from an outside-capture-range sensor 90 of the camera 80, beingdefined by the outside-capture-range DB, and calculates a feature amountof the signal. Then, the situation-expression extraction unit 23 mayextract a situation expression in the situation expression DB, beingassociated with a condition met by the calculated feature amount, as asituation expression outside the capture range of the camera 80.Additionally, when also extracting a situation expression within thecapture range, the situation-expression extraction unit 23 analyzes asignal from a sensor 90 within the capture range of the camera 80defined by the outside-capture-range DB, and calculates a feature amountof the signal. Then, the situation-expression extraction unit 23 mayextract a situation expression in the situation expression DB, beingassociated with a condition met by the calculated feature amount, as asituation expression within the capture range of the camera 80.

The display-method determination unit 24 determines a display method(e.g. a position, a shape, a size, a transmissivity, a movement, and anoverlapping order) on a captured image of the camera 80 with respect tothe situation expression extracted in Step B3, by referring to thesensor positional relation DB (Step B4).

The presentation unit 25 superimposes each situation expressionextracted in Step B3 onto a captured image displayed by the image signalof the camera 80 input in Step A1, in accordance with the display methoddetermined in Step B4, and outputs the captured image superimposed withthe situation expression (Step B5).

When a situation expression based on a sensor 90 within the capturerange is further extracted in Step B3, the presentation unit 25 mayfurther superimpose the situation expression within the capture range,in addition to the situation expression outside the capture range, inStep B5. Alternatively, out of one or more situation expressions outsidethe capture range, the presentation unit 25 may superimpose a situationexpression different from the situation expression within the capturerange.

The above concludes the operation of the information presentation system2.

The operation of the information presentation system 2 will be describedwith a specific example.

It is assumed that the information presentation system 2 has an officespace as an observation space, and includes three cameras 1 to 3, threemicrophones 1 to 3, and two thermometers 1 and 2 as cameras 80 andsensors 90, as illustrated in FIG. 5. It is further assumed that thecamera 2 is applied as the camera 80 according to the present exampleembodiment, and the cameras 1 and 3, the microphones 1 to 3, and thethermometers 1 and 2 are applied as sensors 90. It is further assumedthat the information presentation system 2 holds theoutside-capture-range sensor DB in FIG. 6, the sensor positionalrelation DB in FIG. 8, and the situation expression DB in FIG. 18. It isfurther assumed that, with respect to each point on captured images bythe cameras 1 and 3, a position coordinate in a camera image coordinatesystem of the camera 2 is stored in advance as a conversion table. It isfurther assumed that the information presentation system 2 presetshigher importance to the thermometers 1 and 2 than the other sensors 90.

In this case, first, the signal input unit 11 acquires three types ofimage signals from the cameras 1 to 3, three types of voice signals fromthe microphones 1 to 3, and two types of temperature signals from thethermometers 1 and 2. Then, the signal input unit 11 applies the imagesignal from the camera 2 as an image signal from the camera 80 accordingto the present example embodiment, and applies the other two types ofimage signals, the three types of voice signals and the two types oftemperature signals as signals from the sensors 90 according to thepresent example embodiment (Step B1).

The signal-source-position estimation unit 22 estimates positions ofsignal sources, in accordance with the respective signals from thecameras 1 and 3, the microphones 1 to 3, and the thermometers 1 and 2(Step B2).

The signal-source-position estimation unit 22 applies the positions ofthe sensors 90 (the microphones 1 to 3 and the thermometers 1 and 2)themselves with respect to the signal sources of the voice signals fromthe microphones 1 to 3 and the temperature signals from the thermometers1 and 2, respectively.

Further, the signal-source-position estimation unit 22 performsdetection processing of a human region on the image signals of thecamera 1 and the camera 3 as sensors 90. It is assumed that a humanregion is not detected in the image signal of the camera 3, and a humanregion is detected in the image signal of the camera 1. Specifically, asillustrated in FIG. 19, the signal-source-position estimation unit 22detects a human region with a coordinate (P1x,P1y) as a center at a timet1 with respect to the image signal from the camera 1, and detects thatthe center of the human region moves to a coordinate (P1x′,P1y′) afterΔt. The center coordinates of the human region indicate relativepositions with respect to a center (C1x,C1y) of the captured image ofthe camera 1.

The situation-expression extraction unit 23 refers to theoutside-capture-range sensor DB in FIG. 6, and specifies the cameras 1and 3, the microphones 1 and 3, and the thermometers 1 and 2 asoutside-capture-range sensors 90 of the camera 80 (camera 2). Further,the situation-expression extraction unit 23 specifies the microphone 2as a sensor 90 within the capture range of the camera 80 (camera 2).Then, the situation-expression extraction unit 23 analyzes each signalfrom the specified sensors 90, and calculates a feature amount. Then,the situation-expression extraction unit 23 refers to the situationexpression DB in FIG. 18, and extracts a situation expression associatedwith a condition met by the feature amount (Step B3).

More precisely, with respect to the human region detected as illustratedin FIG. 19 in the image signal from the camera 1 as anoutside-capture-range sensor 90, the situation-expression extractionunit 23 calculates a moving speed thereof, |P1x−P1x′,P1y−P1y′|/Δt.|P1x−P1x′,P1y−P1y′| denotes a distance between the coordinates (P1x,P1y)and (P1x′,P1y′). Further, “I” denotes division. It is assumed that thecalculated moving speed is greater than THw2. Then, thesituation-expression extraction unit 23 selects a text image “Striding”indicating walking as a situation expression associated with a condition“THw2≤ movement of the human region” met by the signal from the camera1, in accordance with the situation expression DB in FIG. 18. A humanregion is not detected from a signal of the camera 3 as anoutside-capture-range sensor 90, and therefore a feature amount meetinga feature amount condition set in the situation expression DB in FIG. 18is not detected.

Further, the situation-expression extraction unit 23 calculates atemporal change with respect to a temperature signal from thethermometer 1 as an outside-capture-range sensor 90. It is assumed that,consequently, a temperature change greater than or equal to THh isobserved within Δt. Then, the situation-expression extraction unit 23selects a text image “Fire” as a situation expression associated with acondition “THh<temperature variation” met by the temperature signal fromthe thermometer 1, in accordance with the situation expression DB inFIG. 18. It is further assumed that a feature amount meeting a featureamount condition set in the situation expression DB in FIG. 18 is notdetected from the signal of the thermometer 2 as anoutside-capture-range sensor 90.

The situation-expression extraction unit 23 performs frequency analysisand sound level measurement on respective voice signals from themicrophones 1 and 3 as outside-capture-range sensors 90 and themicrophone 2 as a sensor 90 within the capture range. It is assumedthat, consequently, a human voice having a sound level greater than THs1is observed within Δt in the voice signal from the microphone 1. It isfurther assumed that a human voice having a sound level less than THs1is observed within Δt in the voice signal from the microphone 3. Then,the situation-expression extraction unit 23 selects a text image“Buzz-buzz” as a situation expression associated with a condition “voiceand THs1≤ sound level” met by the voice signal from the microphone 1, inaccordance with the situation expression DB in FIG. 18. Further, thesituation-expression extraction unit 23 selects a text image“Whispering” as a situation expression associated with a condition“voice and 0<sound level<THs1” met by the voice signal from themicrophone 3. It is further assumed that a feature amount meeting afeature amount condition set in the situation expression DB in FIG. 18is not detected from the signal of the microphone 2 being a sensor 90within the capture range.

Next, with respect to each sensor 90 (the camera 1, the temperaturesensor 1, and the microphones 1 and 3) for which a situation expressionis selected, the display-method determination unit 24 acquires apositional relation between a position of the signal source and thecamera 80 by referring to the sensor positional relation DB in FIG. 8.Then, the display-method determination unit 24 determines a displaymethod of each situation expression, in accordance with the positionalrelation (Step B4).

More precisely, with respect to the center coordinate of the humanregion of the camera 1 illustrated in FIG. 19, the display-methoddetermination unit 24 obtains a position coordinate in the camera imagecoordinate system of the camera 2 by referring to the aforementionedconversion table. Consequently, as illustrated in FIG. 20, the humanregion transitions from a position coordinate p (−p21x,p21y) to a pointp′ (−p21x′,p21y′) within Δt in the camera image coordinate system of thecamera 2. Then, the display-method determination unit 24 determines anintersection of a line connecting a center coordinate (C2x,C2y) of acaptured image of the camera 2 and the point p, and a boundary of thecaptured image of the camera 2, as a first display reference point e1.Further, the display-method determination unit 24 determines anintersection of a line connecting the center coordinate (C2x,C2y) of thecaptured image of the camera 2 and the point p′, and the boundary of thecaptured image of the camera 2, as a second display reference point e2.Then, the display-method determination unit 24 sets regions withinpredetermined ranges from the respective display reference points e1 ande2 (e.g. rectangular regions E1 and E2) as tentative display positions.Additionally, the display-method determination unit 24 determines aregion including the tentative display positions E1 and E2 as thedisplay position of the situation expression of the camera 1. Then, thedisplay-method determination unit 24 determines alternate display on therectangular regions E1 and E2 as a movement of the situation expressionof the camera 1.

Further, in FIG. 20, a position of the signal source of the thermometer1 (a position of the thermometer 1 in this case) is a coordinate(−h21x,h21y) in the camera image coordinate system of the camera 2.Then, the display-method determination unit 24 determines anintersection of a line connecting the center coordinate of the capturedimage of the camera 2 and the position coordinate of the thermometer 1,and the boundary of the captured image of the camera 2, as a displayreference point f of the situation expression regarding the thermometer1. Then, the display-method determination unit 24 determines a regionwithin a predetermined range from the display reference point f (e.g. arectangular region F) as the display position of the situationexpression of the thermometer 1.

Further, in FIG. 20, positions of the signal sources of the microphones1 and 3 (positions of the microphones 1 and 3 in this case) arecoordinate (−m21x,m21y) and a coordinate (m23x,m23y), respectively, inthe camera image coordinate system of the camera 2. Then, thedisplay-method determination unit 24 determines an intersection of aline connecting the center coordinate of the captured image of thecamera 2 and the position coordinate of the microphone 1, and theboundary of the captured image of the camera 2, as a display referencepoint g of the situation expression regarding the microphone 1. Further,the display-method determination unit 24 determines an intersection of aline connecting the center coordinate of the captured image of thecamera 2 and the position coordinate of the microphone 3, and theboundary of the captured image of the camera 2, as a display referencepoint h of the situation expression regarding the microphone 3. Then,the display-method determination unit 24 determines regions withinpredetermined ranges from the respective display reference points g andh (e.g. rectangular regions G and H) as respective display positions ofthe situation expressions of the microphone 1 and the microphone 3.

Further, in FIG. 20, with respect to each of the situation expressionsregarding the sensors 90 (the camera 1, the thermometer 1, and themicrophones 1 and 3), display-method determination unit 24 determines asize thereof, in accordance with preset importance. In this case, a sizeof the situation expression of the thermometer 1 set with higherimportance is determined to be larger than sizes of the respectivesituation expressions of the camera 1, and the microphones 1 and 3.

Further, in FIG. 20, with respect to the display position determined foreach of the situation expressions regarding the sensors 90 (the camera1, the thermometer 1, and the microphones 1 and 3), the display-methoddetermination unit 24 adjusts the position, in accordance with adistance between the captured image center of the camera 2 and theposition of each signal source. Specifically, a situation expression ofa sensor 90 closer to the captured image center of the camera 2, interms of distance, is adjusted to be at a position closer to the frameof the captured image of the camera 2.

Further, it is assumed that the display-method determination unit 24 seta flag displaying a balloon image for specifying a direction of a signalsource.

The presentation unit 25 superimposes each situation expression of thecamera 1, the thermometer 1, and the microphones 1 and 3 onto thecaptured image displayed by the image signal of the camera 2 as thecamera 80, in accordance with the display method determined asillustrated in FIG. 20, and outputs the captured image superimposed withthe situation expressions (Step B5). Since a situation expression is notextracted from the microphone 2 as a sensor 90 within the capture range,the presentation unit 25 superimposes the situation expressions outsidethe capture range, and does not superimpose a situation expressionwithin the capture range. FIG. 21 illustrates an example of an outputcaptured image. In FIG. 21, a captured image by the camera 2 isschematically illustrated.

For example, in FIG. 21, the presentation unit 25 superimposes the textimage “Striding” of the situation expression of the camera 1 onto thedetermined rectangular regions E1 and E2 alternately, for a certainperiod of time. Further, the presentation unit 25 superimposes the textimage “Fire” of the situation expression of the thermometer 1 onto thedetermined rectangular region F in a larger size than the othersituation expressions. Further, the presentation unit 25 superimposesthe text images “Buzz-buzz” and “Whispering” being the respectivesituation expressions of the microphone 1 and the microphone 3 onto thedetermined rectangular regions G and H, respectively. Further, since theflag displaying a balloon image as a display method is set in Step B4,the presentation unit 25 displays each situation expression with aballoon image, and sets a starting point of the balloon at thecorresponding display reference point.

The above concludes the description of the specific example.

Next, an effect of the second example embodiment of the presentinvention will be described.

When presenting a captured image by a camera, the informationpresentation system according to the second example embodiment of thepresent invention is able to more sufficiently convey a situationoutside a capture range in a space including the capture range.

The reason is as follows. According to the present example embodiment,the signal input unit acquires an image signal from the camera and asignal from a sensor as inputs, and the signal-source-positionestimation unit estimates a position of the signal source, in accordancewith the signal from the sensor. Then, the situation-expressionextraction unit extracts a situation expression outside a capture rangecorresponding to a condition met by a feature amount of the signal fromthe sensor outside the capture range, by use of theoutside-capture-range sensor DB and the situation expression DB. Then,by referring to the sensor positional relation DB, the display-methoddetermination unit calculates a positional relation between the capturerange and the position of the signal source, and determines a position,a shape, a size, a transmissivity, a movement, an overlapping order, andthe like as a display method of the situation expression on a capturedimage, in accordance with the calculated positional relation. Then, thepresentation unit superimposes the situation expression onto thecaptured image, in accordance with the determined display method, andoutputs the captured image superimposed with the situation expression.

The present example embodiment may store a situation expressionassociated with a combination of pieces of information about a pluralityof signals in the situation expression DB, and extract a situationexpression by the situation-expression extraction unit, in accordancewith a combination of pieces of information about a plurality ofsignals. Consequently, the present example embodiment is able to conveyto an observer a situation expression more precisely representing asituation outside the capture range.

The present example embodiment may determine a display method such as asize of a situation expression by the display-method determination unit,in accordance with importance of the sensor or the situation expression.Consequently, the present example embodiment is able to moreintelligibly convey to an observer a situation of high importanceoutside the capture range.

The present example embodiment may determine a display method such as aposition and an overlapping order of each situation expression by thedisplay-method determination unit, in accordance with a distancerelation between the capture range and a signal source. Consequently,the present example embodiment is able to more intuitively convey to anobserver a positional relation between a plurality of signal sources.

When the display-method determination unit determines to superimposeonly a situation expression extracted from a sensor in a predeterminedrange, the present example embodiment is able to efficiently convey anatmosphere and a situation of a specific part an observer prefers tocheck.

The present example embodiment may determine a display method of eachsituation expression by the display-method determination unit, inaccordance with a temporal change of a position and a feature amount ofa signal source. Consequently, the present example embodiment is able toreadily convey to an observer a degree of change in a situation outsidea capture range and how an event outside the capture range (or withinthe range) propagates to a situation within the capture range (oroutside the range).

Further, in addition to a situation expression outside the capturerange, the present example embodiment may superimpose a situationexpression within the capture range onto the captured image by thepresentation unit. Consequently, the present example embodiment is ableto convey to an observer whether or not atmospheres and situationswithin the capture range and outside the capture range are similar.

Further, out of one or more situation expressions outside the capturerange, the present example embodiment may superimpose a situationexpression different from a situation expression within the capturerange by the presentation unit. Consequently, the present exampleembodiment is able to convey to an observer an approximate locationoutside the capture range where a situation different from a situationthat can be viewed and heard occurs.

Further, even when a position and an observation direction of the cameraor a sensor change, the present example embodiment determines a displaymethod of a situation expression, in accordance with the change, andtherefore is able to more precisely convey to an observer a situationoutside the capture range.

An example of observing an indoor office space as illustrated in FIG. 5,according to the aforementioned second example embodiment of the presentinvention, has been mainly described. In this case, the informationpresentation system according to the present example embodiment isapplicable to a use for conveying to and sharing with an observer (at aphysically remote location, for example) an atmosphere and a situationof an office space being an observation target. In addition, the spacebeing an observation target of the information presentation systemaccording to the present example embodiment is not limited to an indoorspace but may also be, for example, an outdoor space.

Further, the information presentation system according to the secondexample embodiment of the present invention is also applicable to asupervisory system supervising an anomaly in a space. In this case, theinformation presentation system may detect a signal from a sensorindicating an anomaly in the space such as an alarm bell, a scream suchas “Thief,” a “gunshot,” and detection of a human region during aspecific time period. In this case, the information presentation systemmay extract a situation expression indicating an anomaly such as “Alarmbell,” “Thief,” and “Bang,” in accordance with the detected information,and display the situation expression on a captured image in asuperimposed manner. In this case, when presenting an outdoor or indoorsupervisory video image being a supervisory target, the present exampleembodiment is able to convey to a supervisor a situation of anomalyoccurrence outside a capture range in the space to be supervised.

An example of prestoring the situation expression DB, the sensorpositional relation DB, and the outside-capture-range sensor DB in astorage apparatus, according to the second example embodiment of thepresent invention, has been mainly described. In addition, theinformation presentation system according to the present exampleembodiment may be configured to make an inquiry to an external apparatusstoring equivalent information to the various types of databases.

An example of each functional block in the information presentationapparatuses according to the respective aforementioned exampleembodiments of the present invention being implemented by a CPUexecuting a computer program stored in a storage apparatus or a ROM hasbeen mainly described. Without being limited thereto, the respectivefunctional blocks, in part or in whole, or a combination thereof, may beimplemented by dedicated hardware.

A functional block of the information presentation apparatuses accordingto the respective aforementioned example embodiments of the presentinvention may be implemented in a distributed manner on a plurality ofapparatuses.

The operations of the information presentation apparatuses according tothe respective aforementioned example embodiments of the presentinvention, being described with reference to the respective flowcharts,may be stored in a storage apparatus (storage medium) of a computer as acomputer program according to the present invention. Then, the CPU mayread and execute such a computer program. In such a case, the presentinvention is composed of a code of such a computer program or a storagemedium.

The respective aforementioned example embodiments may be implemented incombination as appropriate.

The present invention has been described with the aforementioned exampleembodiments as exemplary examples. However, the present invention is notlimited to the aforementioned example embodiments. In other words,various embodiments that can be understood by a person skilled in theart may be applied to the present invention, within the scope thereof.

This application claims priority based on Japanese Patent ApplicationNo. 2014-125866 filed on Jun. 19, 2014, the disclosure of which ishereby incorporated by reference thereto in its entirety.

REFERENCE SIGNS LIST

-   -   1, 2 Information presentation system    -   10, 20 Information presentation apparatus    -   11 Signal input unit    -   12, 22 Signal-source-position estimation unit    -   13, 23 Situation-expression extraction unit    -   14, 24 Display-method determination unit    -   15, 25 Presentation unit    -   80 Camera    -   90 Sensor    -   1001 CPU    -   1002 RAM    -   1003 ROM    -   1004 Storage apparatus    -   1005 Peripheral-equipment connection interface    -   1006 Display apparatus

The invention claimed is:
 1. An information presentation apparatuscomprising: a processor configured to: receive an image signal from acamera and a signal from a sensor; estimate a position of a signalsource based on the signal from the sensor; extract information on asituation expression expressing a situation outside a capture range ofthe camera based on the signal from the sensor, the situation expressionhaving a shape, a transmissivity, a movement and a size depending on acontents of expression; determine a display position of the situationexpression on a captured image displayed by the image signal, based on apositional relation between the position of the signal source and thecapture range of the camera; superimpose the situation expression ontothe captured image in accordance with the display position, and outputthe captured image superimposed with the situation expression; and whena plurality of sensors are located at different positions and aplurality of situation expression related to each of sensors areextracted and overlap at least partially on the captured image,determine an overlapping order of the plurality of situation expressionon the captured image based on a distance between the camera and thesensors.
 2. The information presentation apparatus according to claim 1,wherein the processor extracts the situation expression related to thesignal by referring to the situation expression associated in advancewith information about a signal obtainable from the sensor.
 3. Theinformation presentation apparatus according to claim 2, wherein theprocessor extracts the situation expression related to a condition metby a feature amount obtained by analyzing the signal by referring to thesituation expression associated in advance with a condition related to afeature amount of the signal obtainable from the sensor.
 4. Theinformation presentation apparatus according to claim 1, wherein theprocessor extracts the situation expression outside the capture range ofthe camera, in accordance with the signal from a sensor predetermined asa sensor outside the capture range of the camera.
 5. The informationpresentation apparatus according to claim 1, wherein, in addition to thesituation expression outside the capture range of the camera, theprocessor extracts a situation expression within the capture range ofthe camera, in accordance with the signal from a sensor predetermined asa sensor within the capture range of the camera.
 6. The informationpresentation apparatus according to claim 5, wherein, in addition to thesituation expression outside the capture range of the camera, theprocessor superimposes the situation expression within the capture rangeof the camera onto the captured image, in accordance with the displayposition, and outputs the captured image superimposed with the situationexpression.
 7. The information presentation apparatus according to claim5, wherein, out of one or more situation expressions outside the capturerange of the camera, the processor superimposes the situation expressiondifferent from the situation expression within the capture range of thecamera onto the captured image, in accordance with the display position,and outputs the captured image superimposed with the situationexpression.
 8. The information presentation apparatus according to claim1, wherein the processor determines the display position of thesituation expression on the captured image, based on a center coordinateof the captured image and a position coordinate of the signal source ina coordinate system of the captured image of the camera.
 9. Theinformation presentation apparatus according to claim 1, wherein thesensor is at least either one of a microphone and another camera. 10.The information presentation apparatus according to claim 1, wherein theprocessor applies a position of the sensor as the position of the signalsource.
 11. The information presentation apparatus according to claim 1,wherein a plurality of sensors are located at different positions and atleast one sensor is a camera, and wherein the processor is furtherconfigured to detect a movement of human using the image signal from thecamera as the sensor and extract the situation expression related to thedetected movement of human and determine a transmissivity of thesituation expression related to the movement of human depending on adistance between the camera in which the captured image is displayed andthe camera as the sensor.
 12. An information presentation methodcomprising: by a computer, receiving an image signal of a camera and asignal from a sensor; estimating a position of a signal source based onthe signal from the sensor; extracting information on a situationexpression expressing a situation outside a capture range of the camerabased on the signal from the sensor, the situation expression having ashape, a transmissivity, a movement and a size depending on a contentsof expression; determining a display position of the situationexpression on a captured image displayed by the image signal, based on apositional relation between the position of the signal source and thecapture range of the camera; superimposing the situation expression ontothe captured image, in accordance with the display method position, andoutputting the captured image superimposed with the situationexpression; and when a plurality of sensors are located at differentpositions and a plurality of situation expression related to each ofsensors are extracted and overlap at least partially on the capturedimage, determining an overlapping order of the plurality of situationexpression on the captured image based on a distance between the cameraand the sensors.
 13. A non-transitory computer-readable storage mediumstoring an information presentation program that enables a computerapparatus to execute: receiving an image signal of a camera and a signalfrom a sensor; estimating a position of a signal source based on thesignal from the sensor; extracting information on a situation expressionexpressing a situation outside a capture range of the camera based onthe signal from the sensor, the situation expression having a shape, atransmissivity, a movement and a size depending on a contents ofexpression; determining a display position of the situation expressionon a captured image displayed by the image signal, based on a positionalrelation between the position of the signal source and the capture rangeof the camera; superimposing the situation expression onto the capturedimage, in accordance with the display position, and outputting thecaptured image superimposed with the situation expression; and when aplurality of sensors are located at different positions and a pluralityof situation expression related to each of sensors are extracted andoverlap at least partially on the captured image, determining anoverlapping order of the plurality of situation expression on thecaptured image based on a distance between the camera and the sensors.